Nightly Testing Suite #355
Conversation
Set up nightly testing to occur once a week. Run the full test suite, plus recalibrate the test timings for pytest-split if the number of tests have changed and make a PR if necessary
pvandyken
force-pushed
the
maint/gh-actions/nightly
branch
from
545d593 to
3e32e28
Compare
pvandyken
requested review from
akhanf,
kaitj,
myousif9,
Bradley-Karat and
mcespedes99
|
@kaitj, as with the actions refactoring, feel free to use this in the centralized repo, although you'll probably want some automated tests (just to make the workload of generalizing this harder!) |
There was a problem hiding this comment.
This looks good to me workflow wise. Yeah, testing is always a bit iffy with actions I find, but nothing that can't be fixed!
Re: Reusable action - I've been thinking around this since I started doing something similar with the testing / codecov with Afids. I've been a bit hesitant to add testing into the centralized repo primarily due to a lack of standardized testing practices across the different repos. Would love to chat about this some more though and discuss ideas. If moving forward, we settle on a common practice, then I definitely agree that these should be included.
For sure. Actually, the main thing I was referring to was the code to check if the test timings need to be updated, and run if necessary. That's probably a fairly useful and generic chunk of code for any other repos using pytest-split. But yeah, testing is contextual, so even for that workflow, you'd probably need to make the actual test command one of the inputs |
|
Okay, shoot, this is what happens when I don't check my own work before merging. It was supposed to run the full test suite and that seems to be missing, and there's some debug code still present. I'll make a follow-up PR |
pvandyken
added
the
skip_changelog
Add a nightly testing action to be run once per week. This will include running all the tests, plus, it checks whether the number of tests has changed since last calculation of runtimes, and if so, recalculates the runtimes for
pytest-splitand makes a PR.As always with gh-actions, getting this right will involve some dart throwing, so I'll leave this in draft until everything seems to work.Hopefully this works. Its hard to fully replicate the behaviour of it running automatically on the main branch, but we can repair any further issues as they come I think